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Whether a Program is Research Based? 


NRCLD developed this brief to help you understand the terms used in your son’s or 
daughter’s school. This understanding, in turn, can help you determine whether to 
support instructional changes proposed for your son or daughter. 


B ell bottoms, the Macarena dance, Beanie 
Babies — each is a fad that has come and 
gone and may even turn up again. School 
program development, too, can be subject to fads; 
unfortunately, educational fads don’t necessarily 
deliver improved teaching and student achieve- 
ment. 

The benefit of selecting educational programs 
based on proven research is that they can deliver 
what is promised if they are implemented as intend- 
ed. These research-based educational programs are 
like the medicine that your physician prescribes. 
Improvement in the medical condition requires that 
the medicine be taken as prescribed. These educa- 
tional programs have been tested, have produced 


desirable results, and then have been retested to 
make sure the same results happen again. The term 
“research-based” appears in connection with many 
practices and programs. The questions that have 
to be asked about these “research-based” claims 
are, first, “Are they really based on scientific prin- 
ciples?” and, second, “How well were these studies 
done?” 

To better equip yourself to answer these ques- 
tions, you will need to understand some of the ter- 
minology associated with scientific research. Re- 
search studies vary widely in how scientifically they 
were conducted. You need to understand research 
methodology before giving support and making 
any commitment to instructional changes. 


Methodology 

Quality research design starts with 
a review of what other experts 
have done in the field. Then, 
researchers decide the study’s 
methodology’ — how the study 
will be done, who is going to 
participate, where the study will 
be conducted, what procedures 
will take place, what questions 
will be answered, and how the 
results will be judged. 


Variables 


Researchers focus on variables, 
those factors that change on purpose 
within a research study and those 
factors that the researcher hopes will 
not change or affect the results of the 
study. 

The independent variable is the 
one that is changed on purpose by 
the researcher. An independent vari- 
able might be a learning program, a 
new intervention, or a way of teach- 
ing. The independent variable is also 
kn own as the “cause.” 

The effect, or the part of the 


study that appears, disappears, or 
changes as a result of the interven- 
tion (independent variable) is the 
dependent variable, often a measure 
of performance. The effect may be 
contaminated when a third, typically 
unmeasured and unwanted, variable 
enters the process. This confound- 
ing variable could be age, interest, 
motivation, teacher experience, time 
of day, or other factors. Researchers 
try to recognize and control for these 
confounding variables to the great- 
est extent possible. 
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Quality Results 


After results of the study are pro- 
duced, observed, and recorded, 
researchers analyze them to see 
whether any predictable patterns, as- 
sociations, or differences appear and 
what the results actually mean. The 
quality of this interpretation depends 
on the adequacy and appropriateness 
of the evidence. 

Some of the results’ quality 
stems from reliability. For example, 
a student tested with a reliable as- 
sessment would get a similar score 
if the same or similar test were given 
again under the same conditions. In- 
terobserver reliability refers to how 
similarly two people separately rate 


the same behaviors. If one observer 
records a certain behavior as occur- 
ring, the second observer also should 
record that as happening. The higher 
the level of interobserver reliability 
that a study has, the more believable 
the study results. 

Another aspect of evidence qual- 
ity is validity. Validity, or the degree 
to which a score represents what it 
was intended to measure, can be as- 
sessed by answering the following 
two questions: “Did the measure- 
ment system accurately measure 
what it claimed to measure?” and 
“Did the results provide an answer 
to the research question?” 


Other Quality Measures 


Experimental 

Control 

Experimental control, identify- 
ing what caused change in the 
dependent variable, is another 
often-used term. For example, if 
students participating in a study 
designed to teach them a reading 
skill do well on this skill when, 
and only when, they had received 
instruction on it, this is a demon- 
stration of experimental control. 


Replication 

One of the best ways to check a 
study’s validity is to have it done 
again — replication. If you or 
others follow the same process 
and produce similar results, then 
the study results are more likely 
valid. One way to do this is to test 
how well the innovation can be 
replicated in the school. It’s not 
enough to think of the innovation 
as a framework or an approach 
or to use just parts of it. The 
replication should be done using 
the same methods, because if 
they are changed much, then the 
results will change. 


How else can you determine the 
quality of a “research-based” claim? 
Journals, which are the publica- 
tions in which researchers publish, 
typically are peer-reviewed, which 
means that each submitted article is 
examined by fellow researchers who 
analyze the scientific progress. If re- 
search lacks the necessary rigor, the 
journal will not publish the study. 
So, if a study appears in a peer-re- 


viewed journal, most of the judging 
has been done for you. 

Another way to examine the re- 
search quality is to talk to research- 
ers directly and question them, 
especially on how their approach 
would work in your school. You also 
can talk to other researchers and 
ask them their opinion of a study’s 
promised results. 


Although understanding how research is done well involves much more than this quick summary, 
this brief provides a good start. Knowing the basic process and some of the terms used in scientific 
studies will help you better analyze interventions used in your son’s or daughter’s school. 
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